Multi-spectral imaging is important in a variety of applications, including astronomical research, agriculture, archeology, geology, quality control, and surveillance, as well as various medical and military applications. In the context of this document, multi-spectral imaging refers to any imaging technique that simultaneously samples two-dimensional images in, at least, two distinct predefined spectral ranges. In the field spectral ranges are commonly referred to as colors, and may be of any spectral width, may be overlapping or nested, and may lie anywhere in the optical radiation band ranging from infrared (IR) through visible (VIS) to ultraviolet (UV). Near infra-red (NIR) refers to the non-visible part of the electromagnetic spectrum with wavelengths just longer than those of visible red light. In the context of this document, light generally refers to the optical radiation band, and includes the visible spectral range (VIS) and non-visible spectral ranges (IR and UV). Examples of multi-spectral imaging according to this definition include, but are not limited to true-color video imaging, green-red-infrared imaging, and imaging techniques using multiple infrared wavelengths.
Techniques for multi-spectral imaging to capture digital images are known in the art. A popular example is color photography, in which red, green, and blue (“RGB”) color separations are recorded for each image and are recombined to generate a “true color” representation of the scene. Where a series of images is captured, each image is also referred to as a frame. Multi-spectral imaging devices are readily available, including still cameras and video cameras, to capture multi-spectral images of a scene. In the context of this document, a scene is an area or location of interest of which images are being captured. A dynamically changing scene includes a single area of interest in which the contents are changing, moving an imaging device to capture images of a plurality of areas of interest, and a combination of changing content and moving the imaging device.
Multi-spectral imaging that includes spectral ranges outside of the visible spectrum, typically either infrared or ultraviolet regions, allows extraction of additional information from a scene that is not visible to the human eye. For example, where different parts of a scene have similar reflectivity in the visible range but exhibit different reflectivities at IR wavelengths. Depending on the intended application, multi-spectral imaging may have any number of distinct channels from two upwards. Multi-spectral imaging typically refers to no more than dozens of distinct channels, and a larger number of channels are commonly referred to as hyper-spectral imaging.
For display to a human user, the information from various non-visible spectral ranges is commonly mapped into visible colors, producing what is referred to as a “false color” or “synthetic color” image. In the case where a multi-spectral image includes colors from the non-visible spectral range, the multi-spectral image is referred to as a synthetic color image.
A color filter array (CFA), or color filter mosaic (CFM), is a mosaic of color filters placed over the pixel sensors of an image sensor to capture color information. Color filters are needed because typical photosensors detect light intensity with little or no wavelength specificity, and therefore cannot separate color information. The color filters filter the light by spectral range, such that the separate filtered intensities include information about the color corresponding to the filter. The spectral range of a photosensor is commonly referred to as the color, or colors, provided by the photosensor. A combination of color filter mosaic and photosensor is referred to as a mosaic color photosensor. A popular color filter mosaic is the Bayer filter, which gives information about the intensity of light in red, green, and blue (RGB) spectral regions. U.S. Pat. No. 3,971,065 to Bryce E. Bayer for Color Imaging Array teaches a CFA for arranging RGB color filters on a square grid of photosensors that is 50% green, 25% red, and 25% blue, also known as RGGB. A Bayer CFA used with a photosensor is known as a Bayer mosaic color photosensor. In the raw image data from a mosaic color photosensor, each pixel is filtered to record only one of the filter colors, hence the image data from each pixel cannot fully determine color. To obtain a full-color image, various demosaicing algorithms can be used to interpolate a set of complete color values for each pixel.
Additional background information can be found in US patent application 20070145273 to Edward T. Chang for High-Sensitivity Infrared Color Camera, which teaches a CFA including 2×2 blocks of pixels of one red, one blue, one green and one transparent pixel, in a configuration intended to include infrared sensitivity for higher overall sensitivity.
Multi-spectral imaging devices can include filters and mirrors to separate light into multiple colors. Multiple photosensors each capture one or more colors of the separated light. One type of filter that can be used is a dichroic filter. A dichroic filter, thin-film filter, or interference filter is a very accurate color filter used to selectively pass light of a specified range of colors while reflecting other colors. By comparison, dichroic mirrors and dichroic reflectors tend to be characterized by the color(s) of light reflected, rather than the color(s) passed.
U.S. Pat. No. 7,138,663 to Nikon Corporation for Color Separation Device of Solid-State Image Sensor, teaches placement of dichroic mirrors over a triplet of photoreceptors. Specific wavelengths of light are separated and passed to specific photoreceptors designated to record red, green, and blue wavelengths. This system emulates three-CCD imaging systems with a single array.
A digital image photosensor inherently has a limited dynamic range. If too much light reaches the photosensor, the pixels of the photosensor reach saturation and fail to provide further image data (information about the corresponding spectral range of the sensor for the scene being captured). If, on the other hand, too little light reaches the photosensor, no image data will be recorded, or the image data will be spread between a relatively low number of intensity levels, resulting in loss of information or poor quality of the image. The image data is normally kept within the dynamic range of the photosensor by appropriate adjustment of the duration of exposure and/or other parameters affecting the sensitivity of the photosensor. This adjustment may be performed optically, for example by a mechanical or electro-optical shutter deployed in the optical system, or electronically, for example by controlling the electrical signals to the image sensor array which define the integration time, also known as exposure time, of the pixel sensors. The adjustment is typically performed collectively for all of the colors or spectral ranges.
Depending on the application and specific circumstances of a scene, an exposure adjustment may result in non-optimal use of the dynamic range of a sensor for one or more colors when the exposure is adjusted for all colors to avoid over-exposure of a particular color. By way of example, when a color video camera is turned towards a scene such that significant part of the scene is a bright blue sky, the short exposure time necessitated to avoid saturation in the blue color channel may result in loss of important information visible in the red and green color separations.
There is therefore a need for methods and devices for sampling multi-spectral images of a dynamically changing scene, where the exposures of photosensors are independently dynamically adjusted.